Skip to content

feat(mcp): add lexical capability ranking#2573

Closed
reachjalil wants to merge 3 commits into
different-ai:devfrom
reachjalil:feature/mcp-capability-ranking
Closed

feat(mcp): add lexical capability ranking#2573
reachjalil wants to merge 3 commits into
different-ai:devfrom
reachjalil:feature/mcp-capability-ranking

Conversation

@reachjalil

@reachjalil reachjalil commented Jul 8, 2026

Copy link
Copy Markdown
Collaborator

Warning

Queue audit (2026-07-17): returned to draft — do not merge this revision. The lexical-ranking idea is still potentially valuable, but the capability corpus and source contracts have changed since this branch. Rebuild it as a small current-dev ranker/benchmark change with relevance fixtures and latency evidence instead of resolving the stale integration patch.

Manager summary

This replaces source-specific additive keyword scoring with one dependency-free lexical ranker for the full Den capability corpus. REST, administrator, External MCP, marketplace, and skill candidates are merged and ranked on the same 0–100 response scale.

The public contract is unchanged: /mcp/agent still exposes only search_capabilities and execute_capability, and execution/authentication rules remain owned by each source.

What changed

  • Shared tokenization, conservative stemming, stopword handling, synonym expansion, rarity, coverage, adjacency, noise filtering, and near-match suggestions.
  • Candidate builders for REST, administrator, External MCP, marketplace, and skills, followed by one global rerank.
  • Actionable needs_connection and error rows remain discoverable even when ordinary noisy matches are suppressed.
  • Source metadata is preserved so results remain explainable.
  • All-stopword queries short-circuit without unnecessary provider work.

Safety and runtime behavior

  • No embeddings, vectors, LLM calls, or new dependencies.
  • Execute paths, grants, tenant boundaries, credentials, and connection health behavior are unchanged.
  • Current upstream limits remain authoritative: at most 16 external connections, concurrency 4, lifecycle deadline, and structured diagnostics.
  • Network discovery cost is unchanged; this PR only ranks the candidates returned by the existing sources.

Evidence

  • Quality benchmark improved the checked corpus from 10/13 (77%) to 11/13 (85%), including invite a teammate and store a note vocabulary gaps.
  • REST-catalog benchmark: 35,000 searches over 127 capabilities averaged about 0.61 ms per search.
  • 48 focused ranking, catalog, marketplace, External MCP, administrator, organization, and route tests passed.
  • Den API typecheck passed.
  • GitHub Linux/macOS tests, Den/inference artifact builds, i18n, and Helm validation passed on the current head; there are no review threads or requested changes.

Reviewer focus

  1. Ranking weights/noise floor and the small curated synonym map.
  2. Global ordering across sources, especially needs_connection rows.
  3. Preservation of authorization, diagnostics, fanout limits, and execute behavior.

@vercel

vercel Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
openwork-landing Ready Ready Preview, Comment, Open in v0 Jul 13, 2026 6:54pm
1 Skipped Deployment
Project Deployment Actions Updated (UTC)
openwork-app-i7ru Skipped Skipped Jul 13, 2026 6:54pm

@vercel

vercel Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

@reachjalil is attempting to deploy a commit to the Different AI Team on Vercel.

A member of the Team first needs to authorize it.

@reachjalil reachjalil changed the title Draft proposal: Add lexical capability ranking feat(mcp): add lexical capability ranking Jul 8, 2026
@vercel
vercel Bot temporarily deployed to Preview – openwork-app-i7ru July 13, 2026 18:54 Inactive
@reachjalil
reachjalil marked this pull request as ready for review July 13, 2026 18:55
@reachjalil
reachjalil marked this pull request as draft July 17, 2026 08:22
@reachjalil

Copy link
Copy Markdown
Collaborator Author

Superseded by #2966, rebuilt on current dev with the valuable lexical relevance fixtures preserved and current capability contracts intact.

@reachjalil reachjalil closed this Jul 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant